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1.0  INTRODUCTION 


This  document  is  the  Final  Technical  Report  (Contract  Data  Requirements  List  [CDRL]  A002) 
for  contract  number  DAAH01-98-C-R040  entitled,  “Real  Time  Network  Management.”  This 
was  a  7-month  effort,  which  ran  from  24  November  1997  to  24  July  1998.  Synectics  was  the 
prime  contractor  with  the  State  University  of  New  York  (SUNY)  Institute  of  Technology  as  a 
subcontractor  on  the  effort. 

Synectics  Corporation  submitted  a  proposal  abstract,  in  response  to  DARPA  SBIR  SB972-076 
(Real-Time  Network  Performance  Diagnosis),  for  developing  methods  for  diagnosing  network 
performance  problems  in  real  time.  It  supported  the  dominant  theme  of  the  Phase  I  SBIR  with 
research  into  the  problem  and  development  of  a  prototype. 

The  first  phase  of  the  project  was  to  develop  a  prototype  system  that  included  interactive 
graphics  for  describing  the  components  of  a  local  area  network  (LAN)  and  its  interconnections  to 
other  LANs  or  internetworks.  There  were  three  major  considerations  for  this  development  effort. 

□  The  description  had  to  include  hooks  for  modeling  the  expected  behavior  (bandwidth, 
latency,  queuing,  etc.),  and  for  monitoring  the  runtime  behavior  via  traffic  probes, 
Simple  Network  Management  Protocol  (SNMP),  remote  monitoring,  load 
measurements,  etc. 

□  The  modeled  and  observed  behaviors  had  tot  be  displayed  in  a  lucid  manner  that 
assisted  analysts  and  operators,  and  could  be  easily  modified. 

□  The  LAN  descriptions  must  be  modifiable  and  composable  in  a  simple  fashion. 


2.0  OBJECTIVES 


The  objective  of  our  effort  was  to  develop  methods  for  diagnosing  network  performance 
problems  in  real  time  for  DARPA.  According  to  our  Phase  I  research,  it  is  possible  to  collect 
data  on  the  network  and  morph  it  into  queuing  models  to  produce  information  about  the  network 
and  physical  layers  of  nodes  on  a  network. 

The  program  consisted  of  three  tasks.  In  the  first  task  Synectics  assembled  a  set  of  metrics  that 
formed  an  accurate  network  traffic  model,  identified  mechanisms  for  hooking  well-researched 
mathematical  performance  models  into  software  artifacts  to  allow  the  definition  of  expected 
behavior,  and  identified  deviations  from  such  behavior.  Secondly,  Synectics  described  the 
logical  architecture  of  the  proposed  performance  monitoring  system  as  an  object  model.  Finally, 
Synectics  developed  the  prototype  using  Java,  with  support  from  JMAPI  (Java  Management 
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Application  Programmer’s  Interface)  and  JDBC  (Java  Database  Connectivity),  and  the  object 
model  designed  in  the  second  task. 


2.1  TASK  1  -  MATHEMATICAL  MODELING 


A  networking  environment  is  a  dynamic  entity  encompassing  a  finite  number  of  objects,  each 
functioning  to  operate  within  a  specific  set  of  constraints.  Our  goal  was  essentially  to  monitor 
this  environment,  measure  and  make  inferences  on  its  observables,  derive  the  appropriate  states 
of  interest,  and  report  them  appropriately  in  order  to  provide  a  diagnostic  basis  to  network 
managers  to  draw  their  attention  to  exceptional  and  abnormal  events.  Indeed,  to  appropriately 
accomplish  these  activities  and  reasonably  correctly  in  a  real-time  framework,  we  needed  various 
models  of  our  object  to  depict  both  its  own  state  and  that  of  its  constituents.  A  model  of  the 
entire  object  as  a  single  entity  may  not  be  suitable  at  its  constituent  levels.  Not  only  should  we 
be  able  to  appropriately  model  each  one  of  the  network  objects,  such  as  a  switch  or  a  link,  but  we 
should  also  be  able  to  model  the  entire  network  as  a  single  entity  given  our  models  of  its 
constituents.  Secondly,  a  single  model  of  a  dynamic  entity  may  not  be  adequate.  Even  if  such  a 
model  depicts  only  stationary  states,  as  they  almost  always  do,  one  may  be  forced  to  consider 
different  types  of  models  to  handle  different  equilibrium  situations.  For  instance,  at  a  low  traffic 
load,  we  might  use  one  type  of  model  for  an  entity,  but  when  the  traffic  load  becomes  heavy,  we 
might  have  to  use  another  model  to  more  accurately  depict  the  activity  there.  If  this  is  desirable, 
then  switching  from  one  model  to  another  must  be  triggered  by  the  presence  of  an  event  at  a 
meta-level  of  the  models. 

The  local  states  as  perceived  at  each  object  site  are  derived  from  the  raw  observed  data  (the 
observables)  as  appropriate  aggregates.  The  global  system  state  is  a  list  of  all  aggregate  local 
states  of  the  network  along  with  an  appropriate  set  of  computed  temporal  invariants  based  on 
inter-object  coupling  over  the  local  state  space. 

The  object  network  is  seen  as  a  hierarchical  abstraction  as  indicated  below. 

NETWORK  =  {Nodes,  Links} 

Nodes  =  {Ports,  Switches} 

Link  =  (Physical  connection  between  two  nodes)  ...  (1.0) 

A  local  state  variable  $ubj(t)  is  an  attribute  of  an  object  at  a  time  t,  and  we  assume  that  its  short¬ 
term  time-averaged  value  is  “stable”  and  is  denoted  by  djjfi och(t )  for  a  specific  epoch.  The  long¬ 
term  time  averaged  value  of  this  state  is  the  quantity  (which  converges  to  a  steady-state  value  if 
the  network  converges  to  a  steady  state). 
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dabj  =  —  J  dobjOChd(  epoch  )  ...(1.1) 

nep 

where  nep  is  the  epoch  density  over  the  period  of  observation  on  which  the  long-term  behavior  is 

sought.  Our  reporting  profile  is  envisaged  as  follows.  For  every  object  obj  at  any  arbitrary  time 
t,  the  object  manager  is  ready  to  report  the  following. 

obj  =  (object  ID,  object  class) 
time  =  current_time  t 

state_list(t)  =  d(t)  =  (dj(t),d2(t),  ...  ,dp(t))  .  .  .  (1.2a) 

previous _ state _aggregate( epoch _ number  =  -1 )  =  i? ep  1  =  (&fI , ,  ...  )  .  .  .  ( 1 ,2b) 

over  current  and  previous  epochs  ep0  and  ep_, ,  respectively.  Note  that  state _list(t )  is  always  the 
current  state  of  the  object  in  the  current  epoch  ep0 . 

The  estimated  state  list  at  the  next  epoch  ep+] ,  as  inferred  from  the  current  and  the  past  behavior 
through  an  appropriate  intelligent  estimator  (we  assume  that  we  have  appropriate  model(s) 
necessary  to  compute  this),  look  like 


next  _  state  _aggregate(  epoch_number  =  +1 )  =  f)+!  =  ( ,  Oj1 ,  ....  bj/  )  .  .  .  (1.2c) 

And  finally,  the  object  would  post  its  long  term  average  of  the  state  list  over  all  the  epochs  in  a 
given  time-interval  T  as 

dT  =(dj,i)2>  ■■■>  dp, ...  ,g1G,g2G . gqG  )  ...  (1  -2d) 

In  this  case,  a  state  variable  dM  is  the  long-term  average  list  of  the  state  of  the  object  M,  while 
the  qmG  is  the  long-term  average  of  some  computed  variable 

SmG  =fG(di’d2 .  dp)  ...  (1.2e) 

which  has  a  global  relevance  and  a  semantic  such  as  congestion  level.  We  assume  that,  on 
whatever  model  we  propose  to  project  this  measure,  it  is  operationally  stable  and  observable  over 
time  and  it  consists  of  two  spatially  separated  objects  P  and  Q. 

SmG<t  +  *0~$mG(0  and 
imG(P)  =  imG(Q)  -  -  -  (l-2f) 

Note  that  all  these  states  were  either  derived  from  the  local  observables  and/or  inferred  from  both 
the  previous  and  the  current  states.  The  local  observable  at  an  object  switch  may  be  minimally 
the  number  of  messages  and  requests  queued  and  being  serviced  there,  the  server  service  rate,  the 
arrival  rate  at  the  server  queues,  or  the  service  discipline  (e.g.,  FIFO  [first  in,  first  out],  priority, 
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polling,  etc.).  If  it  were  a  link,  the  local  observable  could  be  its  capacity  (static)  and  its  data 
flow-rate  (dynamic).  For  a  switch,  it  could  also  include  the  fraction  of  packets  (or  cells) 
dropped,  packet  injection  rates  at  the  links  to  which  it  is  connected,  or  the  input/output  queue 
densities  at  the  buffers  to  the  links. 


2.1 .1  Local  Perspective 


We  define  local  perspective  as  a  view  of  a  single  network  object  in  the  context  of  the  entire 
network  in  which  it  is  embedded.  In  this  section  we  will  focus  on  the  state  computation  of  a 
single  node  or  a  link  in  a  network.  We  will  offer  a  low-to-moderate  traffic  description  of  a 
server  node  in  an  M/M/l/k  architecture.  Other  realistic  models  would  be  delivered  in  subsequent 
reports. 

2. 1. 1. 1  M/M/1/k  Model  of  a  Single  Server 

The  environment  seen  from  a  node  appears  as  follows  at  the  network  layer  level. 

Figure  1.  Network  Layer  Level 


The  packet  throughput  at  a  node  K,  XK ,  is  apportioned  between  two  streams.  The  inbound 
stream  moving  toward  the  connecting  stations  and  processes  receives  pXK  of  the  node  traffic. 
The  remaining  traffic  appears  at  the  interface  to  be  injected  into  the  network  as  seen  at  the 
interface.  If  the  loss  at  the  interface  is  lossinl,  then  the  amount  going  into  the  network  is 
(l-lossint  )(l-pX K).  Similarly,  if  a  portion  of  the  incoming  traffic  from  Transmission  Control 
Protocol  (TCP)  and  Internet  Control  Message  Protocol  (ICMP)  layers  is  dropped  at  the  node  K, 
the  actual  traffic  entering  from  above  into  the  node  X  fl  would  be  less  than  what  these  nodes  are 
sending  to  the  node  K. 

For  our  convenience,  let  us  assume  that  the  Jackson  assumption  holds  and  the  node  is  seen  as  a 
finite  buffer  Markovian  server  so  that  it  could  be  modeled  as  an  M/M/l/k  system  (k  being  the 
maximum  number  of  packets  the  buffer  is  allowed  to  hold).  In  this  perspective, 
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The  average  total  traffic  entering  into  the  node,  A  K  =  A  ™ '  +aKX  pkts/sec 
The  average  packet  transmission  rate  at  node  K  =  pK  pkts/sec 
The  buffer  size  at  the  node  =  k 

The  blocking  probability  at  the  node  pb^ock  =  -—r—rPk  given  that  its  total  buffer  volume  is  for  k 

l-pk+l 

£ 

packets.  Here  p  =  — — .  We  also  obtain  the  following  equilibrium  results. 

Mk 

(a)  The  average  number  of  packets  in  the  node  (at  the  queue  and  at  the  server) 

A,  p  (k  +  l)pk+1  .t  ,  k 

Nk=- - j-t-  if  A K  =-,  otherwise 

I- p  l-p  2 

(b)  The  effective  average  arrival  rate  into  the  node  is  A  K(l-pk ) 

(c)  Using,  Little’s  law,  the  expected  response  time  at  the  node  TK  = - — - 

A^i-Pk) 

(d)  Average  node  utilization  rate  is  p(l-pk ) 

(e)  The  node  idle  probability  p0  =  -  ^ 

(f)  Queuing  time  spent  in  the  buffer  =  —  zLLEo 

A(l-Pk) 

For  convenience,  we  would  use  the  notation  A6  to  denote  an  appropriate  time  average  (or  a 
weighted  average)  of  an  observable  6(t)  over  an  observation  period  of  size  T.  We  assume  <5r  to 
be  the  size  of  a  monitoring  interval. 

A9=j\8(et+a-et)dt 

Then,  in  terms  of  our  observables  as  obtained  from  the  Management  Information  Base  (MIB), 
the  following  would  be  realized  for  a  node. 

(a)  The  average  packet  traffic  flow  going  into  the  network,  (l-p)XK  =  17 +if  18 ) 

St 

(b)  The  average  traffic  load  delivered  to  the  network  layer  aKX  =  f((Aifl2+Aifll)/6t) 
where /(.)  has  to  be  determined. 

(c)  The  average  traffic  load  delivered  to  the  upper-layer  from  the  node  pXK  =  —  - 
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(d)  The  average  traffic  load  delivered  from  the  TCP/ICMP  to  the  internet  protocol  (IP) 

layer  of  the  node  k  T  =  -  %  — 

St 


(e)  The  blocking  probability  at  the  IP  level  pk  =  =  l  £—pk 

A  ip  10  ] -pk+I 

2. 1. 1.2  Queuing  models  for  Switches  (Routers),  Nodes,  and  Ports 

We  assume  that,  for  our  switches,  nodes,  and  ports,  these  could  be  described  by  appropriate 
queuing  models  M/M/1,  M/M/l/k,  M/G/l,  etc.,  if  the  server  were  a  single  server  or  could  be 
modeled  as  a  single  server. 


A  typical  node  is  modeled  as  a  tandem  queue.  We  assumed  Jackson’s  assumption  holds  in  the 
sense  that  the  queues  are  separable  even  though  individually  they  might  not  be  M/M/1  queues. 
We  considered  a  node  (or  a  port)  as  a  tandem  queue  served  by  two  sets  of  servers,  the  IP-SAP 
(Service  Access  Point)  and  the  server  at  the  interface,  which  we  simply  called  the  interface 
server.  The  interface  server  transmits  requested  frame-traffic  to  the  network  and  receives  frames 
to  send  up  via  the  data-link  layer  to  the  IP-SAP,  or  the  IP  server.  We  assumed  the  service  rates 
at  the  IP  layer  and  at  the  interface  are  given  by  the  parameters  p1  and  p2 ,  respectively.  For 
convenience,  we  let  Xj=k]  and  X2=k2.  Given  these,  we  defined  traffic  densities  at  the 
servers  asp;  =  k]/p]  and  p2  =  k2//i2.  In  terms  of  these  variables  the  node’s  states  could  be 
obtained  using  the  appropriate  queuing  model. 

One  major  problem  here  was  the  estimation  of  the  IP  server  rate  pl .  There  is  no  appropriate 
SNMP  variable  by  which  to  obtain  this  information.  Yet,  at  such  a  service  site  where  packet 
fragments  are  to  be  reassembled,  the  IP-SAP  has  to  wait  for  those  extreme  cases  where  the 
fragments  do  not  turn  up  until  the  last  moment.  (Some  never  arrived.)  This  was  a  time-intensive 
operation  and  therefore  could  not  be  ignored.  One  could  approximate  the  average,  effective 
packet  service  time  as 

—  - ~~J - *  where  nsr'/assemb  denotes  the  number  of  successful  and  unsuccessful 

(nreassemb  +  nreassemb  ) 

packet  reassembly  operations  within  the  given  time  interval  St .  It  is  reasonable  to  assume  that 
nfeassemb  *s  most  likely  positively  correlated  with  the  total  number  of  fragments  received  at  the  IP 
layer  which,  in  turn,  is  linearly  dependent  on  the  incoming  traffic  rate  k  j .  The  factor  U  appears 
here  as  the  server’s  utilization  factor.  We  could  express  this  as  the  requirement  that 


nreassemb  =  V  nreassemb  >  where  t ]  = total  fragme1{  received_  _  }  Qur  effectjve  service  time  becomes 

total  correct  fragments 

7  USt_ 

^  nreassemb(  /  V ) 
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For  each  node,  we  would  obtain  from  the  SNMP  data: 


(a)  The  average  packet  arrival  rate  into  the  system  A I  pkts/sec  at  the  IP  layer,  and  the 
A  2  frames  per  sec  at  the  interface. 


1 


1 


(b)  The  average  service  time  for  a  packet  —  sec  at  the  IP  layer  and  —  sec  at  the 

Mi  Mi 

interface. 


(c)  The  average  number  of  packets  dropped  at  the  node  per  unit  time  nd pkts  where  the 
index  /  e  { lp_  layer,  Interface }  . 


(d)  The  variance  of  the  inter-arrival  times  a2in  obtained  from  the  past  data  at  both  the  IP 
layer  and  the  interface. 

(e)  The  variance  of  the  service  time  <j2/fJ  obtained  from  the  past  data  at  both  the  IP  layer 
and  the  interface. 


If  nd  =  0 ,  then  we  are  effectively  dealing  with  an  infinite  buffer  queuing  system  (buffer-size  k 
00  )•  If  the  variance  a2I/fl  ~  -2— ,  then  we  contend  that  the  server  is  exponentially  distributed. 


Normally,  we  would  assume  the  arrival  process  to  be  Poisson  distributed  in  which  case  the  inter¬ 
arrival  time  must  be  exponentially  distributed  with  a  variance  a2n  =  2— .  However,  if  we  find 


the  variance  substantially  low,  say,  1/qA2 ,  where  q  is  greater  than  2,  then  we  consider  the 
arrival  process  to  be  Erlang-q  distribution.  At  an  interface,  the  service  time  is  essentially  the 
transmission  time  of  a  frame. 


2. 1. 1.3  Model  Selection  Rules 

(a)  The  default  model  would  be  M/M/1  for  each  queuing  subsystem. 

(b)  The  current  model  would  be  the  last  selected  model  unless  it  is  switched  to  another  one. 

(c)  If  nd  =  0,  then  the  selected  model  would  be  one  of  M/M/1,  M/G/l,  M/Ek/1,  Ek/M/1, 
M/D/1,  or  GFG/1. 

(d)  If  nd  *0 ,  then  the  switched  model  would  be  M/M/l/k.  The  blocking  probability  pk  for 
the  model  would  be  computed  as  pk=nd/  A. 

(e)  Once  the  underlying  model  for  a  node  changes  to  a  new  one,  the  earlier  derived  state 
results  would  not  be  used  to  infer  the  future  results.  Note  that  this  is  valid  only  for 
“derived”  results.  The  observed  or  monitored  variables  would  still  be  inferred  when 
needed  from  past  results. 
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(f)  If  the  variance  of  the  arrival  distribution  is  about  -  for  an  arrival  marked  with  A , 

mA  2 

then  the  selected  model  would  be  Em  for  the  arrival  part.  Similarly,  if  the  variance  of 
the  service  distribution  is  about  — -  for  the  service  marked  with  p ,  then  the  selected 

model  would  be  En  for  the  service  side.  Thus,  a  chosen  model  might  appear  as  Em/En/1 
instead  of  M/G/l  if  both  the  arrival  and  the  service  time  distributions  appear  as 
Erlangan. 

(g)  If  the  length  of  the  service  time  per  entity  is  not  exponentially  distributed  (i.e.,  service 
rate  variance  is  different  from  1/p 2  when  the  expected  service  time  is  1/p ),  we 
would  use  a  general  service  time  distribution  and  switch  to  an  M/G/l  or  GI/G/1  model. 
A  GI  arrival  distribution  would  be  considered  if  the  inter-arrival  times  are  independent 
and  identically  distributed.  We  would  use  the  GI/G/1  type  model,  particularly  in  a 
heavy  traffic  load  situation  when  traffic  intensity  p  is  near  to  1 .0. 

(h)  If  the  service  time  per  packet/frame  is  effectively  a  constant,  then  we  would  use  the 
M/D/1  model.  This  would  be  the  preferred  model  for  ports  in  a  bridge. 

2.1. 1.4  M/M/1  Model 

The  parameters  for  this  model  are: 

□  Arrival  rate  A  pkts/sec  into  a  queue  ,  and 

□  Service  rate  p  pkts/sec  of  the  server. 


These  are  time  averaged  over  an  interval  during  which  the  model  is  assumed  to  be  valid.  Given 
these,  we  would  then  compute  the  following. 

(a)  The  traffic  intensity,  p=—  Erlang 

(b)  The  server  utilization,  Unode  =  p 

(c)  The  probability  that  the  number  of  packets  or  frames  in  the  system  is  no  less  than  n, 
p(  N  >n)=  pn 


(d) 

(e) 


The  average  number  of  packets  or  frames  in  the  system  L  = 


The  average  number  of  packets  or  frames  in  the  queue  Lq  = 


P 

1-p 


1-p 


(f)  The  residence  time  (the  waiting  time)  in  the  system  T  = - 

Pd-P) 
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(g)  The  average  queuing  time  in  the  server  queue  Tq  =  —jy — - 

(h)  The  queuing  time  that  r  percent  of  the  customers  do  not  exceed,  i.e.,  the  rth  percentile 

•  *•  ,  ,  _ ,  (  loop  \ 

nnanin/t  ti —  /  ~  \  _  T  /„  *  \ 


queuing  time  nJr)  =  Tln 


100  -r 


(i)  The  rth  percentile  residence  time  nT(r)  =  T  In 


100 -r 


If  the  average  service  time  is  not  directly  available,  one  could  compute  it  as  a  derived  variable 

2  A 

p, effective  =  T+A  and  peffective  = - y  ,  provided  Lq±0 .  This  may  be  necessary  when  the 

-Lq+^L2q+4Lq 

server  is  perceived  as  a  logical  server  such  as  at  a  higher  level  in  a  protocol  suite. 

It  is  possible  that  even  though  the  computed  p  might  appear  to  be  greater  than  zero,  the  observed 
queue  length  at  a  node  may  be  zero  at  a  specific  time  point.  Note  that  these  are  all  time-averaged 
measures;  they  need  not  always  correspond  to  specific  observation  at  a  given  time  point.  Also, 
all  percentile  formulations  would  yield  negative  values  when  p  is  small;  in  such  events,  all 
negative  values  should  be  reported  as  zero. 

2.1. 1.5  M/M/1 /k  Model 

In  this  case,  the  server  has  a  finite  buffer  of  size  k  to  accommodate  incoming  packets/frames. 
This  would  be  indicated  when  we  observe  packets/frames  that  we  assume  are  discarded  owing  to 
lack  of  buffer.  We  assume,  as  before,  the  arrival  rate  and  the  service  rate  to  be  A  and  p , 
respectively. 

A 

(a)  Define  u  =  — .  The  probability  that  the  system  is  at  state  n  (with  n  packets/frames)  is 

M 

,  (l-u)un  <jP 

given  by  Pn  ='- - j—  ifu<l 

l-u 

=  -l—  if  u  =  1 
k  +  1  J 

In  particular,  probability  that  the  system  is  idle  p0  = 

(b)  Probability  that  a  packet/frame  would  be  discarded  for  lack  of  buffer  space 


( 1  —  u)u 


.  Given  this,  we  could  obtain  the  effective  buffer  size  k  over  a  time 
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interval  as  k 


effective 


“(l-Plc  ) 


,  ifu  <  1 


■1,  ifu  =  1 


(c)  The  actual  arrival  rate  at  which  packets/frames  enter  the  system  Xa  =  (l-pk  M 

(d)  f  he  expected  number  of  packets/frames  in  the  system  L  is 

,  u[l-(k+l)uk  +kuM  ]  .r 
L  - - — -  if  u  <  1 

(l-u)(l-uk+1 ) 

=  4  ifu  =  1 


(e)  The  expected  queue  length  Lq  =  L-(l-p0) 

(f)  The  expected  residence  time  T=  L/ X  a 

(g)  The  expected  queuing  time  Tq  =  Lq/ka 

(h)  The  utilization  of  the  server  U  =(l-pk)u 


2.1. 1.6  M/D/1  Model 

In  this  case,  for  given  k,  p,  p ,  we  obtain  the  following.  Note  that  this  model  results  when  the 
variance  of  the  service  time  at  a  server  is  close  to  zero.  This  would  be  applicable  to  model  ports 
in  a  bridge. 


L  =  p+ 


2(1- p) 


Lq=-^-- 

q  2(1- p) 


T  =  —+ - - - 

P  2p(l-p) 


T  = _ C _ 

9  2p(l-p) 


2.1. 1.7  M/G/1  Model 


For  this  model,  assume  we  have  obtained  the  variance  of  the  service  time  a]  as  well  as  the  mean 
service  time  as  1/p  for  a  server.  Let  C2S  =  p2a2s  (for  an  exponential  server  this  will  equal  1). 


Then, 


L  =  p+ 


pffMch 

2(1-  p) 


Lq  —  L  p 


T  =  —+- 


p  p(l-p)\  2 


T  =T- 

H  1 
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2. 1. 1.8  An  Architecture  for  a  Bridge 

A  bridge  connects  two  or  more  LAN  segments  working  entirely  in  the  link  and  physical  layers. 
In  other  words,  there  is  no  IP  layer  associated  with  a  bridge  and  a  bridge  would  receive  only 
frames,  not  packets.  A  typical  transparent  bridge  connecting  Ethernet  LANs  may  be  viewed  in 
Figure  2. 


Figure  2.  Typical  Bridge  between  LANs 
LAN  3 


pi 


LAN  1 


In  the  above  example,  a  bridge  with  four  active  ports  is  used  to  connect  four  different  LANs.  For 
each  port  Pt  we  envisage  a  double  tandem  queue  model,  as  for  switches  and  nodes  shown  earlier, 
except  that  the  service  rate  for  the  first  queue  would  be  -> °°  •  This  amounts  to  effectively 
discounting  the  first  queue. 

Therefore,  the  queue  at  a  port  Pi  would  be  seen  as  a  single  queue.  Thus  each  port  of  the  bridge  is 
equivalent  to  a  node  as  modeled  earlier.  Diagrammatically,  it  appears  then  as  a  single  queue  (see 
Figure  3). 


Figure  3.  A  Detailed  View  of  a  Port  on  a  Bridge 


From  a  LAN 


To  the  LAN 
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2.1 .2  Global  Perspective 


The  abstraction  at  the  global  network  level  appears  as  follows.  The  global  network  may  be  seen 
as  a  network  belonging  to  a  WAN  (Wide  Area  Network)  class  or  it  may  be  a  LAN  type.  Even 
within  these  two  categories  many  different  possibilities  exist,  each  of  which  may  qualify  as  a 
possible  class  for  the  network  in  question. 

In  this  section,  we  will  highlight  the  global  states  of  networks  pertaining  to  two  common 
categories,  the  WAN  architecture  and  the  IEEE  802.3  LAN  architecture.  In  the  next  report,  we 
would  address  other  common  LAN  architectural  models  and  observe  network  global  states  in 
those  possible  architectures. 

The  model  suite  for  the  global  network  state  computation  would  include  minimally  one  model, 
which  would  be  derived  from  the  individual  local  states  of  the  network  components.  We  should 
recall  that  in  our  design  the  global  network  state  net( t)  is  an  appropriate  abstraction  based  on  the 
local  states.  Thus,  the  global  state  would  depict  mean  network  throughput,  mean  network  delay, 
etc. 

We  assume  the  following.  For  the  time  being,  we  would  approach  the  global  state  through  a 
mean  value  analytical  approach.  We  would  assume  that  the  arrival  rate  A  and  the  total  service 
demand  at  any  server  would  be  appropriately  computed.  Given  this,  we  now  indicate  how  the 
relevant  parameters  for  the  global  state  are  to  be  computed  for  our  two  network  classes,  the 
WAN  and  IEEE  802.3  LANs.  As  indicated  earlier,  the  token  ring  architecture  framework  would 
be  outlined  in  the  next  report. 

2. 1.2. 1  WAN-class  Networks 

In  WANs,  the  network  is  seen  as  a  point-to-point,  store-and-forward  type,  packet-switched 
subnet  over  a  wide  distance.  Let  us  assume  [4]: 

(a)  Number  of  reception/transmission  ports  of  the  entire  system  =  Npom 

(b)  Number  of  switches  in  the  system  =  Nmltch 

(c)  Total  number  of  active  nodes  in  the  system  =  Nports  +  N  switch 

(d)  Mean  throughput  rate  at  a  node  K  =  xK  packets/sec  (reported  for  a  local  object  K) 

(e)  System  throughput  rate  =  x  packets/sec  (to  be  derived) 

(f)  Mean  response  time  (queuing  time  +  service  time  delay)  at  a  node  K  =  E(Rk)  sec 
(reported) 

(g)  Mean  residence  time  (total  delay)  per  packet  in  the  system  =  E(R)  sec  (to  be  derived) 


12 


Synectics  Corporation 


Report  No.  WH97JR00-A002 


(h)  Total  mean  external  arrival  rate  of  packets  at  a  port  K  =  XK  pkts/sec  (reported) 

(i)  Total  traffic  in  the  network  system  =  X  packets/sec  =  pkts/sec  (derived) 

K 

Little’s  law  yields  the  following. 

XE(R)  =  N1E(Rk)Xk  ...(4.1) 

K=1 

Since  packets  are  most  likely  switched  through  more  than  an  one  internal  node,  we  note  that 
Total  system  throughput  x  =  XX,  >  l  ...  (4.2) 


with  the  mean  delay  in  the  network 

N  Switch  X  v  X  X  v  X  v 

E(R)  =  I  -f-  E(RK)  =  —  x  E(RK)  =  VI  —*-E(  Rk  )  ...(4.3) 

K=i  A  AX  X 

where  V  is  the  mean  visit  count  of  nodes  per  packet  (i.e.,  the  number  of  nodes  visited  by  a 
generic  packet).  Note  that  the  mean  delay  per  node  per  packet  over  the  system  is  then 

E(  R„ode )  =  I  -jjr  E(  Rk  )  ...(4.4) 

This  perspective  is  derived  for  the  network  layer.  We  could  similarly  derive  the  global  states  as 
seen  at  the  physical  layer  where  traffic  flow  is  in  bits/sec  rather  than  pkts/sec. 

Also,  let  us  suppose  DK  is  the  total  observed  service  time  received  by  a  generic  packet  at  the 
server  K.  We’ll  identify  a  threshold  demand  D.  Since  XDK  =  U K  <1.0  for  all  servers  supporting 

a  network  throughput  X,  DK  <—  and  the  one  with  the  highest  value  must  act  as  a  bottleneck, 

X 

i.e.,  Dbottle  neck  =  maxj D, }  .  .  .  (4.5).  We  may  either  report  Dbottle  neck  or  any  DK  >  D  as 

potential  problem  parameters. 

2.1. 2.2  IEEE  802.3-class  Networks 

For  this  class  of  networks,  we  would  be  concerned  solely  with  Ethernet-based  LANs  [5].  Instead 
of  modeling  an  Ethernet  via  an  exact  queuing  system,  which  would  be  extremely  difficult  and 
possibly  not  be  worth  the  effort,  we  desire  to  capture  its  behavior  via  a  queuing  network  whose 
parameters  could  be  fine-tuned  to  correlate  actual  observation  with  predicted  values. 

Let  us  consider  the  Carrier  Sense  Multiple  Access  with  Collision  Detection  (CSMA-CD) 
protocol  that  controls  the  behavior  of  such  a  network  at  the  system  level.  A  typical  process 
consists  of  alternating  contention  and  a  frame  transmission  period;  idle  periods  will  not  result  if 


13 


Synectics  Corporation 


Report  No.  WH97JR00-A002 


we  assume  there  is  at  least  one  busy  station  at  all  times.  We  would  later  deviate  from  this 
assumption  by  incorporating  an  appropriate  factor  to  reflect  possible  idle  time  for  the  channel. 
Assuming  a  constant  load  for  the  network  with  an  average  of  n  active  stations  connected  to  it,  we 
obtain  the  following. 

The  maximum  probability  A  that  one  station  will  successfully  acquire  the  channel  is 

A  =  (l-—)n~I  ...(4.6a) 
n 

The  average  length  of  the  contention  interval  between  two  successive  transmissions  is 

C  =-  =  (i--)I~n  ...(4.6b) 

An 

The  channel  efficiency  (utilization)  rate  on  the  average  is 

E,n,‘7Tc’-^L  -•<4-6c) 

AF 

The  average  delay  per  station  is 

D=(F  +  C)(n-])/B  ...(4.6  d) 

where  B  is  the  channel  bandwidth  rate.  One  could  also  post  an  interesting  measure 

C(n)  =  —  E(n)  .  .  .  (4.6e) 

F 

This  is  a  system-level  measure,  which  may  be  of  importance  when  we  want  to  know  how  much 
of  the  maximum  theoretical  capacity  of  the  network  is  actually  used  to  deliver  workload.  Note 
that  individual  stations  connected  to  the  LAN  would  be  modeled  as  shown  in  the  Local 
Perspective,  Section  2.1.1. 


2.2  TASK  2  -  OBJECT  MODELING  FOR  ARCHITECTURE 

2.2.1  Managed  Objects 


For  our  monitor,  every  managed  object  is  either  a  physical  object  like  a  node,  or  a  network  link. 
It  could  also  be  a  logical  object  like  a  model  of  a  switch  or  a  model  of  a  link.  In  fact,  all  possible 
models  in  which  we  might  be  interested  pertain  to  a  model  class,  which  would  be  maintained 
within  the  JMAPI  organization  as  another  managed  entity. 
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For  a  given  network,  the  attributes,  or  current  states  of  the  managed  objects  (ports,  switches,  and 
links)  are  obtained  in  a  virtual  information  store,  termed  the  Management  Information  Base 
(MIB).  In  our  case,  we  are  specifically  concerned  with  the  architecture  at  the  network  and 
physical  layers.  The  SNMP-brokered  objects  that  are  to  be  used  by  JMAPI  are  outlined  below. 
These  are  the  basic  variables  to  be  collected  to  derive  models  at  higher  levels  of  abstractions  (for 
details,  see  RFC  1213  [7]). 

Note  that  the  following  is  only  a  subset  of  the  objects  in  which  we  would  eventually  be 
interested.  These  are  the  objects,  which  are  currently  being  used. 

For  Inbound  Traffic  from  the  Network  to  a  Higher  Laver 

□  ifln Octets  (ifEntry  10)  Description— Total  number  of  octets  in  the  interface  for 
inbound  traffic,  including  the  framing  characters. 

□  iflnUcastPkts  (ifEntry  11)  Description-Total  number  of  sub-network  unicast  packets 
delivered  to  a  higher  level  protocol. 

□  ifUnNUcastPkts  (ifEntry  12)  Description— Total  number  of  broadcast/multicast  sub¬ 
network  packets  delivered  to  a  higher  level  protocol. 

□  iflnDiscards  (ifEntry  13)  Description— Total  number  of  inbound  packets  discarded 
due  to  lack  of  buffer  space. 

□  iflnErrors  (ifEntry  14)  Description— Total  number  of  inbound  packets  discarded  due 
to  errors. 

□  iflnUnknownProtos  (ifEntry  15)  Description— Total  number  of  inbound  packets 
discarded  due  to  unknown  protocol  specification. 

At  Interface 

□  ifSpeed  (ifEntry  5)  Description— The  current  estimate  of  the  interface’s  bandwidth  in 
bits/sec. 

For  Outbound  Traffic 

□  ifOutOctets  (ifEntry  16)  Description— Total  number  of  octets  transmitted  to  the 
network  from  this  interface  (including  framing  characters). 

□  ifOutUcastPkts  (ifEntry  17)  Description— Same  as  for  inbound  traffic  but  now  with 
changed  direction. 

□  ifOutNUcastPkts  (ifEntry  18)  Description— Same  as  for  inbound  traffic,  but  now  with 
changed  direction. 

□  ifOutDiscards  (ifEntry  19)  Description — Same  as  for  inbound  traffic,  but  now  with 
changed  direction. 
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□  ifOutErrors  (ifEntry  20)  Description — Same  as  for  inbound  traffic,  but  now  with 
changed  direction. 

□  ifOutQLen  (ifEntry  21)  Description— The  instantaneous  length  of  the  output  packet 
queue  in  packets. 

For  Inbound  Traffic  from  the  Network  Laver  to  the  TCP  Laver 

□  ipINReceives  (ip  3)  Description— Total  number  of  input  datagrams  received  from 
interface  including  those  in  error. 

□  ipInHdrErrors  (ip  4)  Description— Total  number  of  discarded  input  datagrams  due  to 
header  errors. 

□  ipInAddrErrors  (ip  5)  Description-Total  number  of  discarded  input  datagrams  due  to 
address  errors. 

□  ipInUnknownProtos  (ip  7)  Description— Total  number  of  discarded  input  datagrams 
due  to  unknown/unsupported  protocols. 

□  ipInDiscards  (ip  8)  Description— Total  number  of  discarded  input  datagrams  due  to 
lack  of  buffer  space. 

□  ipInDelivers  (ip  9)  Description— Total  number  of  input  datagrams  successfully 
delivered  to  DP  user  protocol  (including  ICMP). 

Note  that  these  variables  pertain  to  input  packet  traffic  from  a  node  to  its  processes  at  the 
application  level.  This  is  distinct  from  the  input  data  traffic  from  an  interface  (at  the  physical 
layer)  to  its  link  layer. 

For  Outbound  Traffic  (All  Traffic  from  IP  User  to  the  Node  for  Transmission) 

□  ipOutRequests  (ip  10)  Description— Total  number  of  IP  datagrams  sent  to  the  node  for 
transmission. 

□  ipOutDiscards  (ip  1 1)  Description— Total  number  of  outbound  datagram  discards  due 
to  lack  of  buffer  space. 


2.2.2  Model  Architecture 

At  this  stage,  we  noted  the  inadequacies  of  all  network  models  currently  in  use.  As  Kleinrock 
[1]  points  out,  an  exact  mathematical  model  of  the  system,  even  if  we  are  lucky  enough  to 
develop  it  in  the  first  place,  may  not  be  tractable  in  real  situations.  We  always  end  up  with  an 
approximation  even  if  our  model  is  exact.  Secondly,  a  desirable  scheme  may  be  to  abandon 
exact  mathematical  models  but  rely  on  an  approximate  model  to  provide  approximate  but 
sufficiently  satisfactory  solutions  if  they  correlate  well  with  the  predictions  of  actual 
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measurements  [1].  In  order  to  accomplish  this,  it  may  be  necessary  to  learn  the  model 
parameters  for  an  object  through  appropriate  learning  models  based  on  the  observed  pasts  and 
the  performance  of  the  learning  process  itself.  We  note  that  a  single  model  to  depict  the  behavior 
of  a  network  object  may  not  be  tenable  for  all  workload  levels,  for  all  types  of  traffic  and  for  all 
topologies.  A  multimodel  framework  is  essential.  In  view  of  these,  our  model  architecture  is 
proposed  as  follows.  For  a  network  object  K,  the  managed  object  server  provides  a  Managed 
Model  Interface  (MMI)  in  which  a  number  of  applicable  models  for  that  object  would  reside  (see 
Figure  4).  This  would  be  true  for  every  managed  network  object  as  well  as  for  the  model  for  the 
global  state  computation. 


Figure  4. 


A  View  of  Model  Interface 


The  model  switch  would  be  triggered  by  the  learning  module,  which  would  decide  how  to  infer 
an  object  state  and,  given  the  currently  inferred  state,  would  decide  which  kind  of  model  would 
be  chosen  and  its  parameters.  Then  the  managed  object  would  call  that  specific  model  object 
and  run  it  with  the  given  set  of  parameters  for  the  current  session.  What  types  of  models  would 
be  best  suited  for  our  network  objects  was  investigated  next. 


First,  we  attempted  to  settle  the  issue  by  using  the  traditional  Markov  chain  model  assuming  that 
the  stochastic  process  (N(t),  t>0}  is  essentially  Markov  (N(t)  being  the  number  of  requests, 
packets  in  the  system).  Even  though  the  situation  improved  considerably,  the  state-space 
explosion  still  remained  the  dominant  issue  that  was  difficult  to  ignore.  Secondly,  the 
computation  hinged  heavily  on  the  distributions  of  the  various  events  -  the  arrival  or  departure 
processes  did  not  need  to  be  renewal  processes  (independent  and  identically  distributed),  which 
an  embedded  Markov  chain  would  minimally  require.  This  made  it  difficult  to  proceed 
realistically  if  we  needed  to  depart  from  the  renewal  process  assumption. 

Next  we  examined  our  model  from  another  angle.  We  could  provide  a  mean-value  analysis 
(MV A)  of  our  network  environment  based  on  the  observed  and  estimated  means.  One  could,  on 
this  base  model,  add  the  necessary  perturbations,  like  fluctuations  about  the  means,  as  well  as 
ordering  or  scheduling  constraints  or  correlations  among  different  steps  and  different  requests 
within  the  overall  scheme  of  a  typical  Markovian  model.  It  was  then  feasible  to  express  the 
distribution  functions  (e.g.,  number  of  requests  waiting  at  a  node)  in  a  product  form,  effectively 
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achieving  network  decomposition  as  if  each  function  depended  solely  on  the  workload  for  that 
node.  It  was  similar  to  expressing  an  arbitrary  function  as  a  Taylor  series  over  some  elemental 
functions.  This  Jackson  model  [2]  based  on  the  product  form  of  distribution  could  be  used  even 
though  one  could  not  strictly  defend  it  theoretically. 

So  we  proceeded  in  the  following  manner.  Keeping  the  overall  Jackson  model  in  view,  the 
network  would  be  functionally  decomposed  here  but  without  embracing  Jackson’s  assumption. 
We  assumed  that  an  object  was  essentially  separate  from  the  network,  except  for  what  it  received 
and  sent  out  to  the  network.  The  network  is  seen  from  the  object  perspective  as  another  distinct 
object  with  which  it  interacts  continuously  but  the  object  is,  otherwise,  totally  independent  from 
the  other  objects.  The  observed  stochastic  process  embracing  the  object  would  be  viewed 
essentially  as  a  Markov  process  with  its  parameters  appropriately  determined  using  an  intelligent 
learning  system.  Essentially,  the  base  models  would  be  primarily  Markovian.  They  would  be 
queuing-network  models  (M/M/1,  M/G/l,  and  GI/G/1).  Other  candidate  models  included  time- 
dependent  stochastic  Petri-net  and  stochastic  process  algebra  models,  along  with  Fractional 
Brownian  motion-based  and  fractional  ARIMA  models. 

The  emphasis  in  Phase  I  of  our  investigation  was  to  provide  the  system  with  a  workable  structure 
to  use  in  selecting  the  most  appropriate  and  tractable  traffic  model  from  among  a  collection  of 
traffic  models  based  on  currently  observed  real-time  data  and  information  through  collected 
legacy  data.  The  model  architecture  as  proposed  was  the  key  to  our  operation.  This  was 
essential  for  another  important  reason.  If  the  operational  scope  for  our  system  is  to  extend  to  the 
second-generation  Internet  it  must  be  able  to  interface  well  with  gigabit  per  second  networks  as  it 
should  with  megabit  per  second  nets.  In  the  latter  case,  as  pointed  out  in  Kleinrock  [3],  the 
network  is  primarily  bandwidth-driven  whereas  in  gigabit  regime,  it  is  latency  driven.  Obviously 
the  two  require  different  modeling  emphasis  and  both  these  distinctions  should  be  available  in 
the  modeling  suite. 


2.3  TASK  3  -  PROTOTYPE  IMPLEMENTATION 
2.3.1  Installation  of  Software 


The  first  item  of  business  was  the  installation  of  the  software  required  to  support  the  Real  Time 
Network  Management  System  (RTNMS).  The  software  in  question  was  the  Java  Developer’s 
Kit  version  1.1.5  (JDK),  the  JMAPI  version  0.5  beta,  Fast  Forward’s  JDBC  driver  version  1.3, 
and  SNMP  agents  on  every  machine  that  was  to  be  managed. 

The  JDK  was  installed  first  to  facilitate  the  installation  of  the  other  software  components,  all  of 
which  required  the  use  of  the  Java  interpreter.  Then  the  JDBC  driver  was  installed.  This  had  to 
be  installed  prior  to  JMAPI  so  that  the  API  (Application  Programmer’s  Interface)  would  be  able 
to  communicate  with  the  database  server  (Sybase  version  1 1),  which  was  already  installed  on  the 
machine  deemed  the  JMAPI  server.  Next  came  the  installation  of  the  JMAPI  software  itself. 
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Once  JMAPI  was  installed,  it  was  configured  per  the  documentation’s  directions.  One  advantage 
was  that  all  the  software  components  utilized  were  identical  to  the  components  used  in  the 
example  provided  in  the  documentation,  so  configurations  were  almost  identical. 

The  final  requirements  were  the  SNMP  agents  that  allow  the  prototype  to  get  information  about  a 
node.  These  varied  from  machine  to  machine  and  it  was  necessary  to  install  different  agents  for 
the  different  platforms  that  were  to  be  managed.  The  SNMP  agents  were  installed  on 
Windows95,  Windows  NT  4.0,  and  Solaris  2.5  workstations.  The  Windows  NT  machines  have 
an  SNMP  agent  built  into  the  system,  so  it  was  only  a  matter  of  activating  those  agents.  Most 
routers  today  come  with  an  SNMP  agent  built  in  as  well,  including  the  local  router  and  one  of  the 
AFRL  routers. 


2.3.2  Initial  Data  Collection 

Once  the  software  was  installed,  it  was  possible  to  start  using  the  classes  that  JMAPI  provided  to 
collect  data  from  any  machine  that  was  running  an  SNMP  agent.  However,  this  involved 
considerable  time  for  the  researcher  to  become  familiar  with  how  SNMP  worked  as  a  protocol, 
and  to  learn  the  JMAPI  SNMP  classes.  The  very  first  iteration  of  the  data  collector  did  little 
more  than  obtain  the  system  description  of  the  machine  that  was  being  queried. 

During  the  process  of  implementing  the  data  collector,  a  fault  was  discovered  in  how  the  JDK 
1.1.5  worked  with  one  of  the  classes  within  JMAPI.  Between  versions  1.1.4  and  1.1.5  of  the 
JDK,  the  developers  changed  the  way  in  which  the  interpreter  handled  Uniform  Resource 
Locator  (URL)  information.  The  ResourceLocator  class  that  came  with  JMAPI  used  the  method 
found  in  the  1.1.4  version,  which  would  cause  NulIPointerExceptions.  Sun  Microsystems 
provided  an  updated  version  of  the  class,  which  once  installed,  allowed  the  data  collector  to 
function  properly. 

With  a  working,  albeit  infantile  data  collector,  it  was  decided  to  try  to  run  the  application  as  an 
applet.  An  attempt  was  made,  but  failed.  Thinking  this  might  have  something  to  do  with  the 
collector,  focus  was  turned  to  the  demonstration  applets  that  came  as  part  of  the  JMAPI 
distribution.  However,  it  was  not  possible  to  get  any  of  these  to  work  as  an  applet  either.  It  was 
possible  to  run  them  as  applications  only.  So,  in  the  interest  of  continuing  development,  the  idea 
of  running  the  RTNMS  from  a  browser  was  abandoned. 

Work  continued  on  expanding  the  collector.  One  major  hurdle  was  the  way  SNMP  stores 
information  about  interfaces.  The  interfaces  are  stored  within  a  MIB  table,  and  the  conventional 
way  to  retrieve  this  information  is  one  entry  at  a  time,  at  least  in  SNMPvl.  For  purposes  of  this 
program  though,  all  these  data  had  to  be  collected  simultaneously  so  the  change  in  time  could  be 
observed.  After  extensive  experimental  work,  the  solution  was  found.  It  involved  accessing  all 
the  elements  at  once  via  their  entry  number  extension.  Unfortunately,  this  was  not  documented 
by  JMAPI  and  was  discovered  intuitively. 
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Modifications  to  the  data  collector  continued  based  on  analysis  of  the  data  it  was  collecting  and  a 
growing  understanding  of  the  SNMP  variables  that  were  being  observed.  This  is  where  many 
misconceptions  were  revealed. 

It  was  discovered  that  the  queue  length  variable  was  the  length  of  the  queue  within  the  interface, 
and  that  SNMP  had  no  measurement  for  the  IP  queue,  probably  because  there  are  so  many 
potential  ways  to  implement  the  IP  layer.  This  complicated  the  modeling  process,  not  only 
because  single  queue  models  were  to  be  used,  but  also  because  there  was  now  no  way  to  get  the 
queue  information  for  IP. 

It  was  also  found  that  some  of  the  SNMP  variables  collected  were  unnecessary  while  some  that 
were  initially  overlooked  had  to  be  retrieved.  The  number  of  routing  table  entries  that  are 
removed  from  the  routing  table  was  deemed  unimportant;  it  was  first  thought  that  this  variable 
measured  some  form  of  packet  dropping.  It  had  to  do  with  the  source  and  destination  routes 
instead.  All  of  the  possible  errors  that  occurred  within  the  interface  and  IP  layers  were  not  being 
collected.  These  errors  were  needed  for  more  accurate  models;  the  number  of  discarded  packets 
due  to  buffer  overflows  was  not  sufficient. 

It  was  also  at  this  stage  that  a  rudimentary  prediction  algorithm  was  implemented  to  test  its 
accuracy.  Samples  showed  that  the  algorithm’s  performance  varied  based  on  the  type  of  network 
activity.  Adjusting  the  weight  of  the  previously  predicted  and  actual  readings  showed  it  was 
possible  to  get  better  results  with  a  variable  weight.  So  the  average  time  error  over  the  set  of  the 
collected  data  was  used  to  determine  how  to  adjust  the  weight  automatically.  This  scheme  was 
used  in  the  final  prototype. 


2.3.3  Creation  of  JMAPI-managed  Objects 


With  the  data  collection  proceeding,  focus  was  shifted  toward  how  the  data  were  to  be  stored. 
This  is  where  JMAPI’s  ability  to  communicate  with  a  database  came  into  play.  Actually,  JMAPI 
requires  that  it  be  tied  into  a  database  before  it  will  run  properly  and  it  stores  data  in  the  database 
via  the  ManagedObject  class.  JMAPI  comes  with  an  entire  hierarchy  based  on  ManagedObject. 
However,  this  hierarchy  was  being  completely  rewritten  at  Sun  and  they  advised  subclassing 
ManagedObject  directly. 

Documentation  for  creating  managed  objects  was  also  vague  due  to  JMAPI  still  being  in  its 
infancy.  The  hard  copy  documentation,  which  was  more  of  a  tutorial-type  document,  gave  an 
initial  starting  point,  but  was  outdated  and  much  of  the  code  illustrated  was  deprecated. 
Fortunately,  the  online  reference  had  the  current  class  definitions  and  method  calls.  By  doing  a 
little  experimentation,  it  was  possible  to  create  a  functional  managed  object.  This  first  managed 
object  was  little  more  than  an  integer  value,  but  demonstrated  that  it  was  possible  to  “set  and  get” 
information  from  the  JMAPI  database. 

While  importing  the  managed  object  into  the  database,  it  was  discovered  that  the  JDBC  driver 
installed  was  inadequate.  The  JDBC  driver  used  was  the  company’s  shareware  version  and 
could  only  handle  two  simultaneous  connections  to  the  database.  Consulting  with  Sun  again,  it 
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was  learned  that  JMAPI  requires  5  or  6  connections  when  importing  a  managed  object.  The 
JDBC  driver  company  was  contacted  and  they  sent  a  30-day,  50-connection  version  of  the  driver 
so  testing  could  be  done  to  insure  this  was  the  problem  with  importing  managed  objects.  The 
new  driver  was  installed  and  the  managed  object  was  successfully  imported.  This  led  to  the 
purchase  of  Fast  Forward’s  15-connection  driver. 

A  large  amount  of  traffic  was  observed  on  the  loopback  port  on  the  JMAPI  server  during  Phase  I 
prototype  execution.  After  researching  the  problem,  it  was  found  that  the  JDBC  driver  was 
creating  this  extra  traffic.  FastForward,  a  Type  IV  driver  from  Connect  Software,  is  the  JDBC 
recommended  by  Sun  when  using  JMAPI.  It  provides  direct  access  to  Sybase  SQL  (Standard 
Query  Language)  servers.  FastForward  works  by  directly  transferring  and  receiving  information 
from  Java  to  the  SQL  Server  using  TCP/IP  sockets.  The  format  of  data  passed  back  and  forth  is 
Tabular  Data  Stream  format  (TDS)  [6].  This  may  be  a  reason  why  access  to  the  database  was 
less  than  sufficient.  Therefore  we  may  look  into  other  JDBC  drivers  for  a  more  efficient  way  of 
accessing  data  in  Phase  n. 

Once  it  was  shown  that  the  database  was  working  properly  and  managed  objects  could  be  stored 
there,  design  of  the  managed  objects  ensued.  A  rough  draft  of  all  necessary  managed  objects  for 
the  network  management  system  was  conceived.  At  this  point  more  refinement  of  the 
mathematical  models  and  the  way  SNMP  represents  the  interface  and  IP  layers  was  required. 

One  problem  was  the  fact  that  a  node  may  have  multiple  interfaces,  each  of  these  having  its  own 
queue.  Significant  work  had  to  be  done  to  adapt  the  single  queue  models  to  the  now  multiqueued 
node.  A  queue  length  also  had  to  be  mathematically  derived  for  the  IP  layer. 

The  IP  layer  has  the  potential  to  fragment  and/or  reassemble  packets  before  passing  them  down 
to  the  interface  or  up  to  higher  layer  protocols.  This  affected  comparisons  of  arrival  and  service 
rate  because  the  rates  were  in  terms  of  different  types  of  packets.  So  a  fresh  look  at  how  the  IP 
layer  fragments  and  reassembles  packets  was  required.  A  multiplier  was  added  to  the  model  to 
compensate  for  the  discrepancy  between  the  number  of  packets  into  and  out  of  the  IP  and 
interfaces.  The  multiplier  going  from  the  interfaces  to  the  IP  layer  is  less  than  one.  The 
multiplier  from  IP  to  the  interfaces  is  between  one  and  two.  This  approximates  the  number  of 
packets  that  are  resent  by  the  datalink  layer,  for  which  there  is  no  SNMP  data. 

Another  multiplier  had  to  be  used  to  calculate  the  percentage  of  packets  going  to  each  of  the 
interfaces  from  the  IP  layer.  This  was  used  in  conjunction  with  the  datalink  multiplier  to 
determine  how  much  of  the  IP  output  was  going  to  each  interface.  This  was  required  because 
there  was  no  SNMP  variable  to  indicate  the  number  of  packets  to  a  particular  interface. 

The  complexity  of  how  the  IP  layer  functions  was  particularly  difficult  to  model.  With  IP  both 
fragmenting  and  reassembling  packets,  some  of  the  details  of  how  IP  handles  packets  gets  lost. 
In  addition,  IP  does  not  maintain  a  count  of  the  packets  that  are  lost  because  of  reassembly 
errors.  So  attempts  to  extrapolate  this  information  were  made.  In  addition  to  this,  some  of  the 
machines  being  monitored  represented  the  reassembly  information  differently  than  the  RFC  1213 
[7]  specifies.  The  Sun  workstation  does  not  give  the  number  of  fragments  that  need  to  be 
reassembled;  instead  it  gives  the  number  of  reassembled  packets.  It  is  unknown  why  Sun  would 
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choose  to  do  this  in  their  MIB;  perhaps  it  has  something  to  do  with  their  implementation  of  the 
IP  layer. 


2.3.4  Integration  of  Components 

The  managed  objects  were  implemented  and  loaded  into  the  database.  These  objects  were 
created  the  same  way  normal  tables  in  a  database  would  be  created,  each  one  having  its  own 
primary  and  foreign  keys.  JMAPI  allows  the  use  of  associations  and  relationships  between 
managed  objects,  but  with  the  documentation  being  sparse,  it  was  deemed  easier  to  create  the 
objects  with  keys  and  allow  application  software  to  ensure  integrity  constraints. 

With  all  the  necessary  managed  objects  created  within  the  database,  it  was  time  to  implement  the 
code  to  do  all  the  work. 

The  initial  collector  would  read  in  the  user’s  information  on  a  node,  (name,  IP  address,  node 
type,  and  read  community).  It  would  then  use  that  information  to  query  the  node  to  get  the 
number  of  interfaces  the  node  has,  the  IP  addresses  of  all  those  nodes,  speeds,  etc.,  and  to 
populate  the  appropriate  managed  objects  with  that  information. 

With  this  information,  whole  topologies  were  created,  and  the  user  could  select  which  topology 
he/she  wanted  to  monitor.  In  order  to  monitor  each  node  as  efficiently  as  possible,  the 
monitoring  software  was  written  to  create  a  thread  for  each  node  that  was  being  monitored.  Each 
thread  is  responsible  for  polling  the  SNMP  agent  of  the  node  it  is  monitoring  as  well  as  sending 
the  received  data  to  the  JMAPI  server  to  be  stored. 

Even  though  the  process  of  collecting  data  from  the  SNMP  agents  is  fairly  efficient,  the  process 
of  using  JMAPI  to  store  and  retrieve  such  an  enormous  amount  of  data  is  not.  Even  with  only 
six  nodes,  the  time  it  took  to  collect  and  store  the  data  was  in  the  order  of  minutes.  Most  of  this 
time  was  due  to  the  complex  process  of  using  the  JMAPI  calls  to  perform  the  “sets  and  gets.” 

A  network  administrator  could  then  look  at  all  this  information  with  the  implementation  of  the 
graphical  user  interface  (GUI).  The  GUI  uses  the  Java  Abstract  Windowing  Toolkit  (AWT) 
package  as  well  as  the  PropertyBook  classes  that  come  with  JMAPI.  This  interface  provides  an 
efficient  and  intuitive  way  for  an  administrator  to  access  and  analyze  all  the  information  that  is 
being  collected  about  the  network. 


2.3.5  Explanation  of  Implementation 


The  software  used  included  the  JDK  version  1.1.5  and  the  JMAPI  version  0.5,  both  from  Sun.  In 
addition,  there  was  Fast  Forward’s  JDBC  version  1.3  allowing  JMAPI  to  communicate  with  the 
Sybase  database  version  1 1 .  Sybase  is  used  to  store  all  the  information  about  the  machines  being 
monitored  on  the  network.  SNMP  agents  were  installed  on  multiple  platforms  so  that  the 
information  about  that  machine  could  be  collected. 
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JMAPI  is  a  tool  created  to  aid  in  the  development  of  network  resource  and  management 
applications.  It  simplifies  this  task  by  abstracting  every  aspect  of  a  network  into  something 
known  as  a  managed  object.  Anything  related  to  the  network  is  a  subclass  of  this  managed 
object.  Doing  this  allows  the  developer  to  focus  on  the  intricacies  of  the  network  to  be  managed 
and  not  how  the  information  will  be  stored  in  the  database;  JMAPI  handles  all  these  details. 

Figure  5.  Explanation  of  Implementation 


Initially  collect  information  about  a  node  when  it  is  created  in  JMAPI  database 
(Number  of  Interfaces;  IP  Addresses, etc.) 


The  application  components  sit  on  top  of  the  JMAPI  interface.  The  Initial  Collector  Process  uses 
the  SNMP  classes  that  came  with  JMAPI  to  collect  information  on  a  host  that  is  currently  being 
added  to  the  database.  Once  a  node  has  been  added  to  the  database,  it  can  be  added  to  one  or 
more  topologies  and  monitored.  The  initial  collector  can  also  be  run  from  within  the  GUI. 

The  Topology  Monitor  is  the  main  part  of  the  Management  System.  It  will  take  the  topology  the 
user  wishes  to  monitor  and  will  run  off  a  thread  for  each  of  the  nodes  that  are  within  that 
topology.  Each  thread  will  communicate  with  the  SNMP  agent  on  the  host  the  thread  is 
monitoring.  The  thread  collects  the  necessary  data  and  then  communicates  to  JMAPI  in  order  to 
get  the  information  stored  persistently. 

The  GUI  can  then  access  the  data  via  JMAPI.  The  interface  supports  the  ability  to  graph  time 
series  of  the  collected  data,  as  well  as  display  the  network  and  its  utilization  based  on  the  data. 
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The  Interface  Layer  is  an  abstraction  of  how  the  interface  card  within  a  computer  handles 
incoming  and  outgoing  data.  Data  come  into  an  interface  from  the  network  and  from  higher 
layer  protocols  within  the  protocol  stack.  Data  from  the  network  are  in  the  form  of  octets;  data 
from  above  are  in  the  form  of  packets  (octets  are  the  number  of  bytes).  Packets  are  groups  of 
bytes  representing  some  sort  of  information.  For  SNMP  version  1.0,  the  packets  can  be  Unicast 
or  Non-Unicast.  For  purposes  of  the  RTNMS,  these  are  summed  together.  Focus  is  on  amount 
of  traffic,  not  types  of  traffic. 

Figure  6.  Interfaces  According  to  SNMP  Variables 


(From  IP  Layer  Protocol)  (To  IP  Layer  Protocol) 


(From  Network) 


(To  Network) 


As  stated  before,  traffic  from  the  network  is  in  terms  of  octets.  A  stream  of  octets  makes  up  a 
packet.  This  stream  is  stored  in  the  interface’s  buffer  until  it  can  be  processed  by  the  interface. 
If  too  many  streams  come  into  an  interface  and  the  buffer  is  not  large  enough  to  accommodate 
them,  some  of  the  streams  (packets)  will  be  discarded.  The  SNMP  variable  iflnDiscards  keeps  a 
count  of  the  number  of  packets  that  the  interface  loses  in  this  manner.  When  the  interface  is  able 
to  process  a  packet  in  the  queue,  it  checks  the  packet  for  errors  and  whether  or  not  the  packet  is 
in  an  understandable  protocol.  SNMP  variables  iflnErrors  and  iflnUnknownProtos  count  the 
packets  lost  when  they  do  not  pass  these  tests.  If  the  packet  makes  it  past  this  point,  it  is  sent  to 
the  next  protocol  up  on  the  protocol  stack.  The  packets  that  make  it  are  counted  in  the 
iflnUcastPkts  and  iflnNUcastPkts  SNMP  variables,  depending  on  which  type  of  packet  it  is. 

When  a  packet  comes  into  the  interface  from  the  higher  protocol,  it  is  counted  in  either  the 
ifOutUcastPkts  or  ifOutNUcastPkts,  again  depending  on  the  type  of  packet  it  is.  It  gets  stored  in 
the  same  buffer  in  which  the  input  from  the  network  is  stored  until  the  interface  can  process  it.  If 
the  buffer  overflows,  and  some  of  these  outbound  packets  are  lost,  they  are  counted  in  the 
ifOutDiscards  variable.  If  the  interface  is  processing  the  outbound  packet  and  finds  some  error 
in  the  packet,  it  will  discard  the  packet  and  increment  the  ifOutErrors  variable.  Finally,  all  the 
octets  that  make  up  the  packets  that  do  get  processed  are  counted  in  the  ifOutOctets  variable. 

Since  it  was  necessary  to  know  the  number  of  packets  travelling  between  the  interface  and  the 
network,  they  had  to  be  derived.  The  number  of  packets  coming  into  the  interface  from  the 
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network  can  be  calculated  by  summing  the  number  of  packets  going  to  higher  protocols 
(iflnUcastPkts  and  ifOutUcastPkts),  iflnUnknownProtos,  iflnErrors,  and  iflnDiscards.  Adding 
all  the  good  packets  to  all  the  dropped  packets  should  give  a  count  of  the  total  number  of  packets 
coming  into  the  interface  from  the  network. 

To  calculate  the  packets  that  actually  make  it  out  to  the  network  from  the  interface,  the  number 
of  packets  discarded  (ifOutDiscards)  and  dropped  because  of  errors  in  the  packet  (ifOutErrors) 
were  subtracted  from  all  the  packets  coming  into  the  interface  from  the  higher  protocol 
(ifOutUcastPkts  and  ifOutNUcastPkts).  Subtracting  the  bad  packets  from  the  total  number  of 
packets  should  give  the  number  of  packets  actually  going  out  to  the  network. 

Figure  7.  The  IP  Layer  According  to  SNMP  Variables 


(From  Interfaces)  (From  Higher  Layer  Protocols) 


The  IP  layer  is  much  more  complicated.  Depending  on  which  task  the  machine  is  performing, 
the  IP  layer  may  perform  differently.  If  the  machine  is  a  router,  a  great  deal  of  packet 
forwarding  will  occur.  It  the  machine  is  a  host,  then  no  packet  forwarding  is  allowed  to  occur. 
However,  if  the  host  is  functioning  as  a  gateway,  then  it  will  forward  packets.  This  complex 
behavior,  in  addition  to  the  vagueness  of  some  of  the  SNMP  variables,  led  to  significant 
confusion. 

Figure  7  shows  the  different  directions  packets  can  take  through  the  IP  layer.  The  ipInReceives 
SNMP  variable  counts  all  the  packets  coming  into  the  IP  layer  from  the  interfaces.  This  includes 
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packets  from  all  of  the  machine’s  interfaces.  These  packets  can  either  be  destined  for  the  higher 
protocols  of  the  machine  or  forwarded  out  to  the  network.  The  blue  line  on  the  graph  represents 
the  path  packets  would  take  through  the  IP  if  they  were  meant  for  higher  protocols.  The  yellow 
line  is  the  path  forwarded  packets  would  traverse.  For  hosts,  the  yellow  path  would  be 
nonexistent.  For  routers,  the  blue  path  would  see  very  little  traffic,  though  some  would  be 
present.  SNMP  requests  would  show  up  as  such.  For  a  gateway,  traffic  would  be  possible  on 
both  routes. 

Packets  coming  from  the  higher  level  protocols  enter  the  IP  layer  and  are  counted  via  the 
ipOutRequests  variable.  This  is  shown  on  the  diagram  as  the  green  path. 

The  crux  of  the  problem  is  that  the  IP  layer  can  fragment  packets  being  sent  to  the  interfaces  into 
smaller  packets  and  reassemble  packets  that  are  going  up  to  the  higher  layers  into  larger  packets. 
This  is  all  done  because  some  networks  cannot  handle  larger  packet  sizes  and  it  is  the  IP  layer’s 
responsibility  to  break  those  packets  down  so  they  can  be  transmitted  over  those  less  capable 
networks.  This  also  means  that  the  packet  counters  at  different  points  of  the  IP  layer  are  not 
counting  packets  the  same  way.  For  example,  ipInReceives  counts  the  smaller  packets,  but 
ipInDelivers  counts  the  same  packets  after  they  have  been  reassembled.  As  an  example  a 
quantity  of  little  packets  come  into  IP  and  ipInReceives  counts  them.  Assume  that  no  discards  or 
errors  occur,  and  some  of  the  packets  are  reassembled.  ipInDelivers  will  show  a  smaller  number 
of  packets  going  to  upper  layer  protocols  than  ipInReceives  said  came  into  IP,  even  though  there 
were  no  losses.  This  is  because  some  of  the  smaller  packets  are  consolidated  into  their  larger, 
original  size.  So  ipInDelivers  is  counting  the  fewer,  larger  packets,  while  ipInReceives  is 
counting  the  smaller,  more  numerous  packets. 

For  the  mathematical  models  to  work  properly,  the  inputs  and  outputs  of  the  IP  layer  must  be  in 
the  same  type  of  packets  (small  or  large).  Since  the  smaller  packets  correspond  to  the  packets 
the  interfaces  send  to  and  receive  of  the  IP  layer,  it  was  decided  to  try  to  convert  all  of  the  inputs 
into  the  smaller  packet  size.  For  a  host,  this  was  not  much  of  a  problem,  but  for  routers  and 
gateways,  the  task  was  much  more  difficult. 

When  dealing  with  a  host,  the  yellow  path  in  Figure  7  can  be  eliminated  because  hosts  are  not 
allowed  to  forward  packets.  The  ipOutRequests  could  be  calculated  in  terms  of  small  packets 
using  the  SNMP  variables  like  this. 

Little  Packets  from  Higher  Protocols  =  ipOutRequests  +  (ipFragCreates  -  ipFragOKs) 

This  formula  takes  ipFragCreates,  which  is  the  total  number  of  packets  created  by  the 
fragmentation  algorithm  and  subtracts  ipFragOKs,  which  is  the  number  of  packets  that  were 
successfully  fragmented.  This  results  in  the  number  of  additional  packets  created  due  to 
fragmentation.  When  added  to  ipOutRequests,  it  gives  the  number  of  little  packets  coming  from 
the  higher  protocols. 

SNMP  does  not  have  a  variable  to  determine  how  many  packets  are  being  sent  to  the  interfaces 
from  IP,  so  this  number  is  derived  as  well.  To  get  this  number,  the  SNMP  variables  are  used 
thusly. 


26 


Synectics  Corporation 


Report  No.  WH97JR00-A002 


Little  Packets  to  the  Interfaces  =  ipOutRequests  +  (ipFragCreates  -  ipFragOKs )  - 

ipOutNoRoutes  -  ipOutDiscards 

The  first  part  of  the  formula  is  the  number  of  little  packets  from  the  higher  protocols. 
Subtracting  ipOutNoRoutes  and  ipOutDiscards  gives  the  number  of  small  packets  that  make  it 
through  IP  and  can  be  sent  to  the  interfaces.  These  formulas  only  work  for  machines  that  are 
running  as  hosts. 

Packets  coming  into  the  IP  layer  from  the  interfaces  are  counted  in  terms  of  little  packets 
already,  so  no  derivation  of  small  packets  is  required. 

Packets  going  from  IP  to  the  higher  layer  protocols  are  measured  in  terms  of  reassembled  (big) 
packets.  This  number  can  be  retrieved  from  the  ipInDelivers  variable.  Getting  this  number  in 
terms  of  small  packets  is  problematic.  This  is  because  the  reassembly  algorithm  cannot  count  the 
packets  that  could  not  be  reassembled.  Therefore,  SNMP  can  only  count  the  number  of  times  the 
reassembly  algorithm  incurs  a  failure.  These  failures  in  no  way  reflect  the  number  packets  that 
are  lost  due  to  those  failures.  The  best  way  to  illustrate  why  this  is  the  case  is  to  outline  how  the 
reassembly  algorithm  works. 

When  the  reassembly  algorithm  receives  a  packet  fragment  that  needs  to  be  reassembled,  it 
creates  a  queue  and  sticks  that  packet  into  it.  Packet  fragments  are,  in  their  own  right,  packets, 
only  smaller.  It  will  then  place  additional  fragments  belonging  to  that  packet  into  the  queue. 
When  all  of  the  fragments  have  been  collected,  the  algorithm  will  assemble  the  packet  and  pass  it 
along  to  the  next  higher  protocol.  The  algorithm  will  only  hold  the  packet  fragments  in  the 
queue  for  a  finite  amount  of  time  however.  If  this  time  limit  is  exceeded,  the  algorithm  will  drop 
all  the  fragments  in  the  queue  and  increment  the  ipReasmFails  variable.  The  problem  arises 
when  failures  occur.  There  is  no  way  to  determine  how  many  fragments  were  lost  because  of 
errors  or  how  many  fragments  made  it  through  to  the  reassembly  algorithm  to  become  the  larger 
packets.  For  the  purposes  of  the  prototype,  it  was  assumed  that  no  packets  needed  reassembly. 
Under  this  assumption,  ipInDelivers  would  therefore  be  in  terms  of  packet  fragments.  The  data 
indicated  that  reassembly  was  indeed  a  rare  occurrence  for  the  network  we  were  monitoring.  It 
is  obvious  that  more  work  is  needed  on  this  aspect  of  the  IP  layer. 

Another  problem  was  encountered  in  the  way  some  of  the  monitored  nodes  represented  one  of 
SNMP  variables.  In  most  cases,  we  know  the  number  of  fragments  that  need  reassembly;  the 
count  is  stored  in  the  ipReasmReqds  variable.  Observation  of  the  collected  data  shows  that 
ipReasmReqds  on  the  Sun  workstations  is  a  count  of  the  number  of  whole  packets  that  need 
reassembly,  not  the  number  of  packet  fragments  that  need  reassembly. 

The  number  of  packet  fragments  for  the  Sun  machines  can  be  derived,  as  long  as  the  machine  is 
functioning  as  a  host.  To  get  the  number  of  fragments,  the  following  formulas  were  used. 

totPktln  =  ipInReceives  -  ipInDiscards  -  iplnHdrErrors  -  ipInAddrErrors  -  ipInUnknownProtos 

Packets  needing  reassembly  =  totPktln  -  ipInDelivers  +  ipReasmOKs 
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If  the  machine  is  operating  as  a  gateway,  then  there  is  no  way  to  ascertain  the  number  of  packet 
fragments  needing  reassembly,  because  it  is  not  possible  to  determine  how  many  of  the 
ipInReceives  packets  are  going  to  higher  protocols  and  how  many  are  being  forwarded. 

While  considerable  work  was  done,  particularly  at  the  IP  layer,  to  model  the  way  nodes  behave 
in  terms  of  SNMP  variables,  it  is  evident  that  more  work  is  needed.  More  definitive  methods  for 
calculating  packet  fragments  going  into  and  coming  out  of  the  IP  layer  are  required.  Another 
possibility  might  be  to  look  into  version  2  of  SNMP  or  to  implement  a  customized  agent  to 
retrieve  the  information  required  for  the  mathematical  models. 
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4.0  GLOSSARY 


API 

Application  Programmer’s  Interface 

AWT 

Abstract  Windowing  Toolkit 

CDRL 

Contract  Data  Requirements  List 

CSMA-CD 

Carrier  Sense  Multiple  Access  with  Collision  Detection 

FIFO 

First  In,  First  Out 

GUI 

Graphical  User  Interface 

ICMP 

Internet  Control  Message  Protocol 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 

IP 

Interface  Protocol 

JDBC 

Java  Database  Connectivity 

JDK 

Java  Developer’s  Toolkit 

JMAPI 

Java  Management  Application  Programmer’s  Interface 

LAN 

Local  Area  Network 

MDB 

Management  Information  Base 

MMI 

Managed  Model  Interface 

MVA 

Mean-value  Analysis 

RTNMS 

Real-Time  Network  Management  System 

SAP 

Service  Access  Point 

SNMP 

Simple  Network  Management  Protocol 

SQL 

Standard  Query  Language 

SUNY 

State  University  of  New  York 

TCP 

Transmission  Control  Protocol 

TDS 

Tabular  Data  Stream 

URL 

Uniform  Resource  Locator 

WAN 

Wide  Area  Network 
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